Agricultural plant detection and control system

ABSTRACT

A computing system includes image receiving logic configured to receive image data indicative of an image of a field, ground identification logic configured to identify a first image portion of the image representing ground in the field, image segmentation logic configured to identify a remaining image portion that omits the first image portion from the image, and crop classification logic configured to apply a crop classifier to the remaining image portion and identify a second image portion of the image that represents locations of crop plants in the field. The computing system also includes weed identification logic configured to identify locations of weed plants in the field based on the identification of the first and second image portions and control signal generation logic configured to generate a machine control signal based on the identified locations of the weed plants.

FIELD OF THE DESCRIPTION

The present description generally relates to agricultural machines. Morespecifically, but not by limitation, the present description relates toplant evaluation and machine control using field images.

BACKGROUND

There are many different types of agricultural machines. One suchmachine is an agricultural sprayer. An agricultural spraying systemoften includes a tank or reservoir that holds a substance to be sprayedon an agricultural field. Such systems typically include a fluid line orconduit mounted on a foldable, hinged, or retractable and extendibleboom. The fluid line is coupled to one or more spray nozzles mountedalong the boom. Each spray nozzle is configured to receive the fluid anddirect atomized fluid to a crop or field during application. As thesprayer travels through the field, the boom is moved to a deployedposition and the substance is pumped from the tank or reservoir, throughthe nozzles, so that it is sprayed or applied to the field over whichthe sprayer is traveling.

The discussion above is merely provided for general backgroundinformation and is not intended to be used as an aid in determining thescope of the claimed subject matter.

SUMMARY

A computing system includes image receiving logic configured to receiveimage data indicative of an image of a field, ground identificationlogic configured to identify a first image portion of the imagerepresenting ground in the field, image segmentation logic configured toidentify a remaining image portion that omits the first image portionfrom the image, and crop classification logic configured to apply a cropclassifier to the remaining image portion and identify a second imageportion of the image that represents locations of crop plants in thefield. The computing system also includes weed identification logicconfigured to identify locations of weed plants in the field based onthe identification of the first and second image portions and controlsignal generation logic configured to generate a machine control signalbased on the identified locations of the weed plants.

This Summary is provided to introduce a selection of concepts in asimplified form that is further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. The claimed subject matter is not limited to implementationsthat solve any or all disadvantages noted in the background.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example agricultural sprayer.

FIG. 2 illustrates an example agricultural sprayer.

FIG. 3 is a block diagram of one example of an agricultural sprayingmachine architecture.

FIG. 4 illustrates one example of a plant evaluation system.

FIG. 5 is a flow diagram illustrating an example operation of a plantevaluation system to identify weed plants from image data.

FIG. 6 illustrates an example image of a field.

FIG. 7 is a block diagram showing one example of the architectureillustrated in FIG. 3, deployed in a remote server architecture.

FIGS. 8-10 show examples of mobile devices that can be used in thearchitectures shown in the previous figures.

FIG. 1 is a block diagram showing one example of a computing environmentthat can be used in the architectures shown in the previous figures.

DETAILED DESCRIPTION

The present description generally relates to agricultural machines. Morespecifically, but not by limitation, the present description relates toplant evaluation and machine control using field images.

As an agricultural spraying machine (or agricultural sprayer) traversesa field, it applies a spray of a liquid (e.g., herbicide, fertilizer,fungicide, or other chemical) using nozzles mounted on a boom. Aspraying system, which typically includes a pump that pumps the liquidfrom a reservoir to the nozzles mounted on the boom, is controlled todeliver a target or prescribed application to the agricultural field.For example, in precision spraying applications, the sprayer iscontrolled to deliver the liquid to a precise dispersal area, such asdirectly on a plant (crop or weed), in between plants, or otherwise, ata particular rate so that a target quantity of the liquid is applied tothe dispersal area. Accordingly, precise application of the liquid isimportant in these applications. For example, if an herbicide isunevenly applied or applied to incorrect plants, it is wasted in areasof over-application, and areas of under-application experience reducedweed prevention.

For sake of illustration, image processing is performed in someprecision spraying applications by acquiring images of the field toidentify the locations of the weeds to be sprayed with an herbicide (orother liquid chemical). In some approaches, a weed classifier is trainedwith training data to detect the various types of weeds to be sprayed.However, due to a wide range of possible field scenarios, such as aplurality of different possible weed types, different weed conditions,different lighting, etc., a large collection of images are required totrain the weed classifier. The computing resources and processingrequired to both acquire and process these images, and to train theclassifier, can be quite burdensome and computational expensive. Evenstill, the trained weed classifier may perform poorly during varyingruntime field applications when trying to classify in different lightingconditions and/or when the weeds are damaged by insects, weather, etc.The inaccurate weed identification results in poor spraying performance.

For sake of the present discussion, a “weed” or “weed plant” refers toany non-crop plant identified in the field. That is, it includes planttypes other than crop plants (e.g., corn plants in a corn field)expected to be present in the field under consideration. In the cornfield example, weeds or weed plants include any plants other than cornplants.

The present disclosure provides a plant evaluation and control systemthat acquires images of a field and processes those images to identifyportion(s) of the image that represent ground (i.e., the soil or othernon-plant areas of the field terrain), and omits those identifiedportions from the image to identify a remaining image portion. Thisremaining image portion is processed by applying a crop classifier todetect, from the plants in the remaining image portion, areas of theimage that represent crop. Locations of weed plants represented in theimage are identified based on the identification of the portions of theimage that represent ground and the portions of the image that representcrop, e.g., by omitting the ground and crop portions from the image, andthen correlating the remaining weed plant image portion to thegeographic field locations. This weed location information can beutilized in any of a number of ways, for example by controlling anagricultural sprayer, generating a weed plant map for the field, to namea few.

FIG. 1 illustrates an agricultural spraying machine (or agriculturalsprayer) 100. Sprayer 100 includes a spraying system 102 having a tank104 containing a liquid that is to be applied to field 106. Tank 104 isfluidically coupled to spray nozzles 108 by a delivery system comprisinga set of conduits. A fluid pump is configured to pump the liquid fromtank 104 through the conduits through nozzles 108. Spray nozzles 108 arecoupled to, and spaced apart along, boom 110. Boom 110 includes arms 112and 114 which can articulate or pivot relative to a center frame 116.Thus, arms 112 and 114 are movable between a storage or transportposition and an extended or deployed position (shown in FIG. 1).

In the example illustrated in FIG. 1, sprayer 100 comprises a towedimplement 118 that carries the spraying system, and is towed by a towingor support machine 120 (illustratively a tractor) having an operatorcompartment or cab 122. Sprayer 100 includes a set of traction elements,such as wheels 124. The traction elements can also be tracks, or othertraction elements as well. It is noted that in other examples, sprayer100 is self-propelled. That is, rather than being towed by a towingmachine, the machine that carries the spraying system also includespropulsion and steering systems.

FIG. 2 illustrates one example of an agricultural sprayer 150 that isself-propelled. That is, sprayer 150 has an on-board spraying system152, that is carried on a machine frame 156 having an operatorcompartment 158, a steering system 160 (e.g., wheels or other tractionelements), and a propulsion system 162 (e.g., internal combustionengine).

FIG. 3 illustrates one example of an architecture 200 having anagricultural spraying machine 202 configured to perform a sprayingoperation on an agricultural field. Examples of agricultural sprayingmachine 202 include, but are not limited to, sprayers 100 and 150illustrated in FIGS. 1 and 2. Accordingly, machine 202 can comprise atowed implement or it can be self-propelled. FIG. 3 illustrates thiswith dashed box 204 representing a towing machine, such as a tractorthat is coupled to machine 202 through one or more links 206(electrical, mechanical, pneumatic, etc.).

Machine 202 includes a control system 208 configured to control othercomponents and systems of machine 202. For instance, control system 208includes a communication controller 210 configured to controlcommunication system 212 to communicate between components of machine202 and/or with other machines or systems, such as remote computingsystem 214 and/or machine(s) 215, either directly or over a network 216.Network 216 can be any of a wide variety of different types of networkssuch as the Internet, a cellular network, a local area network, a nearfield communication network, or any of a wide variety of other networksor combinations of networks or communication systems.

A remote user 218 is illustrated interacting with remote computingsystem 214. Remote computing system 214 can be a wide variety ofdifferent types of systems. For example, remote system 214 can be aremote server environment, remote computing system that is used byremote user 218. Further, it can be a remote computing system, such as amobile device, remote network, or a wide variety of other remotesystems. Remote system 214 can include one or more processors orservers, a data store, and it can include other items as well.

Communication system 212 can include wired and/or wireless communicationlogic, which can be substantially any communication system that can beused by the systems and components of machine 202 to communicateinformation to other items, such as between control system 208, sensors220, controllable subsystems 222, image capture system 224, and plantevaluation system 226. In one example, communication system 212communicates over a controller area network (CAN) bus (or anothernetwork, such as an Ethernet network, etc.) to communicate informationbetween those items. This information can include the various sensorsignals and output signals generated by the sensor variables and/orsensed variables.

Control system 208 is configured to control interfaces, such as operatorinterface(s) 228 that include input mechanisms configured to receiveinput from an operator 230 and output mechanisms that render outputs tooperator 230. The user input mechanisms can include mechanisms such ashardware buttons, switches, joysticks, keyboards, etc., as well asvirtual mechanisms or actuators such as a virtual keyboard or actuatorsdisplayed on a touch sensitive screen. The output mechanisms can includedisplay screens, speakers, etc.

Sensor(s) 220 can include any of a wide variety of different types ofsensors. In the illustrated example, sensors 220 include positionsensor(s) 232, speed sensor(s) 234, and can include other types ofsensors 238 as well. Position sensor(s) 232 are configured to determinea geographic position of machine 202 on the field, and can include, butare not limited to, a Global Navigation Satellite System (GNSS) receiverthat receives signals from a GNSS satellite transmitter. It can alsoinclude a Real-Time Kinematic (RTK) component that is configured toenhance the precision of position data derived from the GNSS signal.Speed sensor(s) 234 are configure to determine a speed at which machine202 is traveling the field during the spraying operation. This caninclude sensors that sense the movement of ground-engaging elements(e.g., wheels or tracks) and/or can utilize signals received from othersources, such as position sensor(s) 232.

Control system 208 includes control logic 240, and can include otheritems 242 as well. As illustrated by the dashed box in FIG. 3, controlsystem 208 can include some or all of plant evaluation system 226, whichis discussed in further detail below. Also, machine 202 can include someor all of image capture system 224. Control logic 240 is configured togenerate control signals to control sensors 220, controllable subsystems222, communication system 212, or any other items in architecture 200.Controllable subsystems 222 include a spraying subsystem 244, machineactuators 246, a propulsion subsystem 248, a steering subsystem 250, andcan include other items 252 as well. Spraying subsystem 244 includes oneor more pumps 254, configured to pump material (liquid chemicals) fromtank(s) 256 through conduits to nozzles 258 mounted on a boom, forexample. Spraying subsystem 244 can include other items 260 as well.

Machine 202 includes a data store 261 configured to store data for useby machine 202, such a field data. Examples include field location datathat identifies a location of the field to be operated upon by a machine202, field shape and topography data that defines a shape and topographyof the field, crop location data that is indicative of a location ofcrops in the field (e.g., the location of crop rows), or any other data.

Machine 202 is illustrated as including one or more processors orservers 262, and can include other items 264 as well. As alsoillustrated in FIG. 3, where a towing machine 204 tows agriculturalspraying machine 202, towing machine 204 can include some of thecomponents discussed above with respect to machine 202. For instance,towing machine 204 can include some or all of sensors 220, component(s)of control system 208, some or all of controllable subsystems 222. Also,towing machine 204 can include a communication system 266 configured tocommunicate with communication system 212, one or more processors orservers 268, a data store 270, and it can include other items 272 aswell. As also illustrated in FIG. 3, towing machine 204 can include someor all components of image capture system 224, which is discussed infurther detail below.

Image capture system 224 includes image capture components configured tocapture one or more images of the area under consideration (i.e., theportions of the field to be operated upon by spraying machine 202) andimage processing components configured to process those images. Thecaptured images represent a spectral response captured by image capturesystem 224 that are provided to plant evaluations system 226 and/orstored in data store 274. A spectral imaging system illustrativelyincludes a camera that takes spectral images of the field underanalysis. For instance, the camera can be a multispectral camera or ahyperspectral camera, or a wide variety of other devices for capturingspectral images. The camera can detect visible light, infraredradiation, or otherwise.

In one example, the image capture components include a stereo cameraconfigured to capture a still image, a time series of images, and/or avideo of the field. An example stereo camera captures high definitionvideo at thirty frames per second (FPS) with one hundred and ten degreewide-angle field of view. Of course, this is for sake of example only.

Illustratively, a stereo camera includes two or more lenses with aseparate image sensor for each lens. Stereo images (e.g., stereoscopicphotos) captured by a stereo camera allow for computer stereo visionthat extracts three-dimensional information from the digital images. Inanother example, a single lens camera can be utilized to acquire images(referred to as a “mono” image).

Image capture system 224 can include one or more of an aerial imagecapture system 276, an on-board image capture system 278, and/or otherimage capture system 280. An example of aerial image capture system 224includes a camera or other imaging component carried on an unmannedaerial vehicle (UAV) or drone (e.g., block 215). An example of on-boardimage capture system 278 includes a camera or other imaging componentmounted on, or otherwise carried by, machine 202 (or 204). An example ofimage capture system 280 includes a satellite imaging system. System 224also includes a location system 282, and can include other items 284 aswell. Location system 282 is configured to generate a signal indicativeof geographic location associated with the captured image. For example,location system 282 can output GPS coordinates that are associated withthe captured image to obtain geo-referenced images 286 that are providedto plant evaluation system 226.

Plant evaluation system 226 illustratively includes one or moreprocessors 288, a communication system 290, a data store 292, an imageanalysis system 294, and can include other items 296 as well. Data store292 can store the geo-referenced images 286 received from image capturesystem 224, plant evaluation data generated by system 226, or any otherdata used by system 226 or other machines or systems of architecture200. Communication system 290, in one example, is substantially similarto communication system 212 discussed above.

FIG. 4 illustrates one example of plant evaluation system 226. As shownin FIG. 4, system 226 includes a user interface component 300 configuredto generate user interface(s) 302 having user input mechanism(s) 304 foraccess by a user 306. User 306 interacts with user input mechanisms 304to control and manipulate plant evaluation system 226. For example, user306 can control image analysis system 294, view images 308 stored indata store 292, to name a few. Also, user 306 can view the imageanalysis results and evaluate how to treat the field (or variousportions within the field) based upon the results. Plant evaluationsystem 226 can also generate recommendations for treating various spotswithin the field, based upon the analysis data. This can vary widelyfrom things such as applying more herbicide, applying fertilizer, toname a few. A control signal generator logic 310 is configured togenerate control signals to control items of system 226, or other itemsin architecture 200.

Image analysis system 294 includes image receiving logic 312 configuredto receive images from image capture system 224 and image pre-processinglogic 314 configured to pre-process those images. For example, logic 314includes a shadow corrector 316 configured to perform shadow correctionon the images, illumination normalizer 318 configured to normalizeillumination in the image, image combiner 320 configured to combineimages, and can include other items 322 as well.

Image combiner 320, in one example, is configured to combine a number ofimages into a larger image of the field under analysis. For instance,image combiner 320 can mosaic the images and geo-reference them relativeto ground control points. In order to mosaic the images, geographiclocation information corresponding to each of the images is used tostich them together into a larger image of the field under analysis,which is then analyzed by system 294. Further, the geo-referencing ofimages can be done automatically against the ground control points, orit can be done manually as well.

Geo-referencing logic 324 is configured to geo-reference the images, orcombined images, to locations in the field, spatial analysis logic 326is configured to perform spatial analysis on the images, and spectralanalysis logic 328 is configured to perform spectral analysis on theimages. Spatial analysis logic 326, in one example, obtainspreviously-generated crop location data which provides a geographiclocation of the rows of crop plants (or the plants themselves). Forexample, this can be generated during a planting operation using aplanting machine. Of course, crop location data can be obtained fromother sources as well. In any case, the crop location data can beutilized to identify the crop rows, and thus the areas between the croprows that are expected to be free of crop plants. This can includeidentifying a reference line that corresponds to the center of each croprow along with a margin window around that reference line, for each row.As discussed in further detail below, plants identified between twoadjacent reference lines (and/or margin window) can be assumed to be anon-crop plant (e.g., a weed plant).

Spectral analysis logic 328 performs spectral analysis to evaluate theplants in the images. In one example, this includes identifying areas inthe image that have a spectral signature that corresponds to groundversus plants. For instance, this can include a green/brown comparison.

Image segmentation logic 330 is configured to perform image segmentationon a received image, to segment or divide the image into differentportions for processing. This can be based on ground and/or plant areaidentifications by ground/plant identification logic 332, and cropclassification performed by crop classification logic 334. This isdiscussed in further detail below. Briefly, however, ground/plantidentification logic 332 identifies areas of an image that representground and areas of an image that represent plants, for example usingthe spatial and spectral analysis performed by logic 326 and 328,respectively.

Crop classification logic 334 uses a crop classifier, that can betrained by crop classifier training logic 336. In one example, the cropclassifier is trained using crop training data 337 stored in data store292, or obtained otherwise. The crop classifier is configured toidentify areas in the image that represent crop plants.

Weed identification logic 338 is configured to identify weeds in theimage, based on the image segmentation performed by image segmentationlogic 330. This is discussed in further detail below. Briefly, however,image segmentation logic 330 is configured to identify a weed plantportion of a received image, that omits a first portion of the imagethat represents ground in the field and a second portion of the imagethat represents crop plants. This remaining portion of the image isdetermined to represent weeds. Illustratively, in one example, thisprocess is performed without using a weed classifier or otherwisedirectly identifying weed plants in the image.

The location of the weeds can be stored as crop location data 339, weeddata 340 in data store 292, which can store other items 342 as well.

Image analysis system 294 can also include anomaly detection logic 344and can include other items 346 as well. Anomaly detection logic 344 isconfigured to detect anomalies based on the weed plant image (e.g., theportion of the image remaining after the ground image portion and thecrop image portion have been omitted). Illustratively, a detectedanomaly represents anomalous crop detections. For instance, one exampleof an anomaly is a crop plant detection in an area that is in betweenthe crop rows. This can represent a false positive detection by the cropclassifier, and can be used to re-train the crop classifier to improveits performance.

FIG. 5 illustrates one example of a flow diagram 400 for identifyingweed plants from image data and corresponding machine control. For sakeof illustration, but not by limitation, FIG. 5 will be described in thecontext of plant evaluation system 226 in architecture 200.

At block 402, a crop classifier to be used by crop classification logic334 is selected based on a selected crop. For example, the selected cropcan be selected by operator 230. In one example, the crop classifier isa classifier that is trained by training logic 336 accessing croptraining data 337 to classify portions of an image representing plantsas being crop plants, as opposed to non-crop plants or weeds. Thetraining data can take any of a variety of forms, such as images labeledwith crop data identifying areas of the images that are crop plantsand/or areas of the image that are non-crop plants. For example, totrain a corn plant classifier configured to identify corn plants in aplant image, a set of training images are labeled with identifiers thatidentify the areas of the image that represent crop plants.

At block 404, image data indicative of an image of a field is receivedby image receiving logic 312. As noted above, the image can be obtainedin any of a wide variety of ways. The image can comprise a single imageobtained by a camera, an image within a time series of still images, oran image from a video. Further, the image can comprise a stereo image(represented by block 406), a mono image 408, or other type of image410. Also, the image can be received from an on-board imaging sensor,such as from on-board image capture system 278. This is represented byblock 412. Alternatively, or in addition, the image can be received froma remote source, such as from remote computing system 214. This isrepresented by block 414. Further yet, the image can be received fromanother machine 215. For instance, a UAV that flies over the field priorto a spraying operation acquires images of the field. This isrepresented by block 416.

At block 418, the image is pre-processed. This can include removing orcorrecting shadows using shadow corrector 316. This is represented byblock 420. Also, the image can be processed to normalize illuminationsusing illumination normalizer 318. This is represented by block 422. Ofcourse, the image can be pre-processed in other ways as well. This isrepresented by block 424.

At block 426, one or more portions of the image representing ground inthe field are identified. This can be done in any of a number of ways.For example, this can be done based on colors identified in the image.For instance, image processing is performed using RGB (red-green-blue)color vectors. RGB color data refers to a color model in which red,green and blue light (or signals or data representative thereof) arecombined to represent other colors. Each pixel or group of pixels of thecollected image data may be associated with an image parameter level ora corresponding pixel value or aggregate pixel value. Thus, each pixelstands for one discrete location in the image and stores data for threechannels (red, green, and blue) to represent a certain color. The imageparameter level is an indicator of or a measure of an image parameterthat observed, reflected and/or emitted from one or more objects in anyother portion of one or more objects within the image. A clusteringalgorithm can be configured to cluster pixel values to generate a basecolor vector and/or averaged color vector for the image. Accordingly, atblock 428, a clustering algorithm can be utilized with RGB IRsegmentation to segment the image based on RGB color vectors.

Referring to FIG. 6, which illustrates an example image 500 of a portionof field, block 428 identifies portions 502 as areas of the imagerepresenting ground, determined based on differences in the color duringcolor segmentation. Portions 502 are, in this example, identified ascontaining brown, or threshold shades of brown, and the other areas ofthe image are green or at least beyond a threshold difference from thebrown color of portions 502. Thus, block 426 identifies portions 502 asrepresenting the ground.

Alternatively, or in addition, the portions of the image representingground can be identified using stereo data, such as a point cloud. Thisis represented by block 430 in FIG. 5. For example, as noted above, thestereo data can provide three-dimensional information to distinguish thelocation of the plant material relative to the ground plane. Thus, thestereo data at block 430 can be used to identify areas of the imagerepresenting material that is above the ground plane (e.g., above theground by a threshold distance. Referring again to FIG. 6, block 430identifies the areas of the image generally represented by referencenumeral 504 as representing plants, and these portions of the image areseparated from the ground portions 502. In one example, a remainingimage portion is obtained that omits the ground portions 502.

The image portions representing the ground can be identified in otherways as well. This is represented by block 432 in FIG. 5. At block 434,the ground portion(s) identified at block 426 are separated to obtain aremaining (non-ground or plant) image portion that represents areas inthe field that include plants (both crop and non-crop plants). Theremaining image portion obtained at block 434 omits the ground portions.This can be done in any of a number of ways. For example, imagesegmentation logic 330 can segment or divide the image, and extract thenon-ground portions, to obtain the remaining image portion. In oneexample, the received image, or a copy thereof, is stored in memory andthe remaining image portion is obtained by removing the image data forthe non-ground portions from the memory, so that only the image data forthe remaining image portion remains. This, of course, is by way ofexample only.

At block 436, crop portions in the remaining image portion are detected.Illustratively, the detection is performed by applying a crop classifierto the remaining image portion. As noted above, crop classificationlogic 334 can apply a crop classifier by crop classifier training logic336 using crop training data 337.

The detection of the crop portions at block 436 can be based on locationwithin the image (block 438) and/or based on color within the image(block 440). Of course, the detection can be performed in other ways aswell. This is represented by block 442.

In one example of block 438, logic 334 obtains crop location data 339.As discussed above, crop location data 339 can be obtained from a priorplanting operation, or otherwise, and indicates the locations where cropseeds were planted, which can be indicative of the crop rows, the cropspacings within the rows, etc. Using crop location data 339, logic 334can identify the locations within the image where a crop plant isexpected. Also, using crop location data 339, logic 334 can determinethat plants that appear in between the rows are likely to be non-cropplants.

In one example of block 440, logic 334 looks at the RGB color vectorsfrom a pixel clustering algorithm to determine whether an area of theimage that represents a plant indicates a crop plant or a non-cropplant.

At block 444, a weed (non-crop plant) image is obtained which representsweed portions in the received image. In the illustrated example, theweed image is obtained based on the identification of a first imageportion (i.e., a ground portion identified at block 426) and a secondimage portion (i.e., a crop portion identified at block 436). That is,the weed image a remaining image portion that omits the ground portionsand the crop portions in the image. This can be done by imagesegmentation logic 330 separating the weed image portion from the otherportions, or otherwise.

For sake of illustration, with reference again to FIG. 6, block 436detects image portion 506 as representing crop based on the location ofthat image portion relative to the rows and based on the color of thatimage portion relative to a portion 508 that represents a weed plant.Block 444 obtains a weed image that includes portions 508, 510, 512,514, 516, 518, and 520, and omits the ground portions 502 and cropportions 506.

Referring again to FIG. 5, at block 446, the weed image is analyzed todetect anomalies. As noted above, anomalies can be detected for any of anumber of reasons. In one example, an anomaly is detected based thelocation on the crop rows. This is represented by block 448. Forexample, if a crop portion is detected at block 436 and resides inbetween crop rows, identified based on location at block 438, then ananomaly is detected as the plant in that portion of the image isunlikely to be a crop plant. Anomalies can be detected in other ways aswell. This is represented by block 450. At block 452, logic 338dynamically updates or tunes the crop classifier based on the detectedanomalies. The updated crop classifier can be re-applied by returning toblock 436.

At block 454, the weed portions identified at block 444 are correlatedto their respective areas of the field. These areas of the field areidentified as containing weeds. In one example, weed identificationlogic 338 generates geographic coordinates for each separate field areathat has been identified as containing weeds, and stores this data asweed data 340.

At block 456, control signal generator logic 310 generates a controlsignal based on the identified field areas. This control signal controlsone or more systems or machines in architecture 200 in any of a varietyof ways.

For example, control signal generator logic 310 can generate a controlsignal to control spraying subsystem 244 to apply a liquid chemical tothe identified field areas. This is represented by block 458. In anotherexample, the control signal generator logic 310 can control a weed mapgenerator 345 to generate a weed map that identifies locations of theweeds on a map of the field.

Alternatively, or in addition, the control signal can controlcommunication system 290 to send the weed data or weed map to a remotemachine or system. This is represented by block 462. For instance, theweed data can be sent to another spraying machine, remote computingsystem 214, etc.

In one example, the control signal controls a display device, such asinterfaces 228 and/or 302 for operator 230 or user 306. This isrepresented by block 464. Of course, the machine control signal can begenerated to control other items in architecture 200. This isrepresented by block 466.

It can thus be seen that the present system provides a number ofadvantages. For example, but not by limitation, performing weedidentification using a remaining image portion, remaining after omittingground and crop portions, increases the speed and efficiency in theprocessing. Further, applying a crop classifier to identify crops fromthe plant portions of the image reduces the computational burden andexpensive. Further, actions can be taken based upon the weedidentification to save chemicals, to save time, and to otherwise improvethe agricultural operations.

It will be noted that the above discussion has described a variety ofdifferent systems, components and/or logic. It will be appreciated thatsuch systems, components and/or logic can be comprised of hardware items(such as processors and associated memory, or other processingcomponents, some of which are described below) that perform thefunctions associated with those systems, components and/or logic. Inaddition, the systems, components and/or logic can be comprised ofsoftware that is loaded into a memory and is subsequently executed by aprocessor or server, or other computing component, as described below.The systems, components and/or logic can also be comprised of differentcombinations of hardware, software, firmware, etc., some examples ofwhich are described below. These are only some examples of differentstructures that can be used to form the systems, components and/or logicdescribed above. Other structures can be used as well.

The present discussion has mentioned processors, processing systems,controllers and/or servers. In one example, these can include computerprocessors with associated memory and timing circuitry, not separatelyshown. They are functional parts of the systems or devices to which theybelong and are activated by, and facilitate the functionality of theother components or items in those systems.

Also, a number of user interface displays have been discussed. They cantake a wide variety of different forms and can have a wide variety ofdifferent user actuatable input mechanisms disposed thereon. Forinstance, the user actuatable input mechanisms can be text boxes, checkboxes, icons, links, drop-down menus, search boxes, etc. They can alsobe actuated in a wide variety of different ways. For instance, they canbe actuated using a point and click device (such as a track ball ormouse). They can be actuated using hardware buttons, switches, ajoystick or keyboard, thumb switches or thumb pads, etc. They can alsobe actuated using a virtual keyboard or other virtual actuators. Inaddition, where the screen on which they are displayed is a touchsensitive screen, they can be actuated using touch gestures. Also, wherethe device that displays them has speech recognition components, theycan be actuated using speech commands.

A number of data stores have also been discussed. It will be noted theycan each be broken into multiple data stores. All can be local to thesystems accessing them, all can be remote, or some can be local whileothers are remote. All of these configurations are contemplated herein.

Also, the figures show a number of blocks with functionality ascribed toeach block. It will be noted that fewer blocks can be used so thefunctionality is performed by fewer components. Also, more blocks can beused with the functionality distributed among more components.

FIG. 7 is a block diagram of one example of the architecture shown inFIG. 3, where machine 202 communicates with elements in a remote serverarchitecture 550. In an example, remote server architecture 550 canprovide computation, software, data access, and storage services that donot require end-user knowledge of the physical location or configurationof the system that delivers the services. In various examples, remoteservers can deliver the services over a wide area network, such as theinternet, using appropriate protocols. For instance, remote servers candeliver applications over a wide area network and they can be accessedthrough a web browser or any other computing component. Software orcomponents shown in FIG. 3 as well as the corresponding data, can bestored on servers at a remote location. The computing resources in aremote server environment can be consolidated at a remote data centerlocation or they can be dispersed. Remote server infrastructures candeliver services through shared data centers, even though they appear asa single point of access for the user. Thus, the components andfunctions described herein can be provided from a remote server at aremote location using a remote server architecture. Alternatively, theycan be provided from a conventional server, or they can be installed onclient devices directly, or in other ways.

In the example shown in FIG. 7, some items are similar to those shown inFIG. 3 and they are similarly numbered. FIG. 7 specifically shows thatsystem 226 and data store 292 can be located at a remote server location552. Therefore, agricultural machine 202 accesses those systems throughremote server location 552.

FIG. 7 also depicts another example of a remote server architecture.FIG. 7 shows that it is also contemplated that some elements of FIG. 3are disposed at remote server location 552 while others are not. By wayof example, data store 292 can be disposed at a location separate fromlocation 552, and accessed through the remote server at location 552.Alternatively, or in addition, system 226 can be disposed at location(s)separate from location 552, and accessed through the remote server atlocation 552.

Regardless of where they are located, they can be accessed directly byagricultural machine 202, through a network (either a wide area networkor a local area network), they can be hosted at a remote site by aservice, or they can be provided as a service, or accessed by aconnection service that resides in a remote location. Also, the data canbe stored in substantially any location and intermittently accessed by,or forwarded to, interested parties. For instance, physical carriers canbe used instead of, or in addition to, electromagnetic wave carriers. Insuch an example, where cell coverage is poor or nonexistent, anothermobile machine (such as a fuel truck) can have an automated informationcollection system. As the agricultural machine comes close to the fueltruck for fueling, the system automatically collects the informationfrom the machine or transfers information to the machine using any typeof ad-hoc wireless connection. The collected information can then beforwarded to the main network as the fuel truck reaches a location wherethere is cellular coverage (or other wireless coverage). For instance,the fuel truck may enter a covered location when traveling to fuel othermachines or when at a main fuel storage location. All of thesearchitectures are contemplated herein. Further, the information can bestored on the agricultural machine until the agricultural machine entersa covered location. The agricultural machine, itself, can then send andreceive the information to/from the main network.

It will also be noted that the elements of FIG. 3, or portions of them,can be disposed on a wide variety of different devices. Some of thosedevices include servers, desktop computers, laptop computers, tabletcomputers, or other mobile devices, such as palm top computers, cellphones, smart phones, multimedia players, personal digital assistants,etc.

FIG. 8 is a simplified block diagram of one illustrative example of ahandheld or mobile computing device that can be used as a user's orclient's hand held device 16, in which the present system (or parts ofit) can be deployed. For instance, a mobile device can be deployed inthe operator compartment of agricultural machine 202 (or 204) or asremote computing system 214. FIGS. 9-10 are examples of handheld ormobile devices.

FIG. 8 provides a general block diagram of the components of a clientdevice 16 that can run some components shown in FIG. 3, that interactswith them, or both. In the device 16, a communications link 13 isprovided that allows the handheld device to communicate with othercomputing devices and under some embodiments provides a channel forreceiving information automatically, such as by scanning. Examples ofcommunications link 13 include allowing communication though one or morecommunication protocols, such as wireless services used to providecellular access to a network, as well as protocols that provide localwireless connections to networks.

In other examples, applications can be received on a removable SecureDigital (SD) card that is connected to an interface 15. Interface 15 andcommunication links 13 communicate with a processor 17 (which can alsoembody processors or servers from previous FIGS.) along a bus 19 that isalso connected to memory 21 and input/output (I/O) components 23, aswell as clock 25 and location system 27.

I/O components 23, in one example, are provided to facilitate input andoutput operations. I/O components 23 for various embodiments of thedevice 16 can include input components such as buttons, touch sensors,optical sensors, microphones, touch screens, proximity sensors,accelerometers, orientation sensors and output components such as adisplay device, a speaker, and or a printer port. Other I/O components23 can be used as well.

Clock 25 illustratively comprises a real time clock component thatoutputs a time and date. It can also, illustratively, provide timingfunctions for processor 17.

Location system 27 illustratively includes a component that outputs acurrent geographical location of device 16. This can include, forinstance, a global positioning system (GPS) receiver, a LORAN system, adead reckoning system, a cellular triangulation system, or otherpositioning system. It can also include, for example, mapping softwareor navigation software that generates desired maps, navigation routesand other geographic functions.

Memory 21 stores operating system 29, network settings 31, applications33, application configuration settings 35, data store 37, communicationdrivers 39, and communication configuration settings 41. Memory 21 caninclude all types of tangible volatile and non-volatilecomputer-readable memory devices. It can also include computer storagemedia (described below). Memory 21 stores computer readable instructionsthat, when executed by processor 17, cause the processor to performcomputer-implemented steps or functions according to the instructions.Processor 17 can be activated by other components to facilitate theirfunctionality as well.

FIG. 9 shows one example in which device 16 is a tablet computer 600. InFIG. 9, computer 600 is shown with user interface display screen 602.Screen 602 can be a touch screen or a pen-enabled interface thatreceives inputs from a pen or stylus. It can also use an on-screenvirtual keyboard. Of course, it might also be attached to a keyboard orother user input device through a suitable attachment mechanism, such asa wireless link or USB port, for instance. Computer 600 can alsoillustratively receive voice inputs as well.

FIG. 10 shows that the device can be a smart phone 71. Smart phone 71has a touch sensitive display 73 that displays icons or tiles or otheruser input mechanisms 75. Mechanisms 75 can be used by a user to runapplications, make calls, perform data transfer operations, etc. Ingeneral, smart phone 71 is built on a mobile operating system and offersmore advanced computing capability and connectivity than a featurephone.

Note that other forms of the devices 16 are possible.

FIG. 11 is one example of a computing environment in which elements ofFIG. 3, or parts of it, (for example) can be deployed. With reference toFIG. 11, an example system for implementing some embodiments includes acomputing device in the form of a computer 710. Components of computer710 may include, but are not limited to, a processing unit 720 (whichcan comprise processors or servers from previous FIGS.), a system memory730, and a system bus 721 that couples various system componentsincluding the system memory to the processing unit 720. The system bus721 may be any of several types of bus structures including a memory busor memory controller, a peripheral bus, and a local bus using any of avariety of bus architectures. Memory and programs described with respectto FIG. 3 can be deployed in corresponding portions of FIG. 11.

Computer 710 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 710 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media is different from, anddoes not include, a modulated data signal or carrier wave. It includeshardware storage media including both volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical disk storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canbe accessed by computer 710. Communication media may embody computerreadable instructions, data structures, program modules or other data ina transport mechanism and includes any information delivery media. Theterm “modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal.

The system memory 730 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 731and random access memory (RAM) 732. A basic input/output system 733(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 710, such as during start-up, istypically stored in ROM 731. RAM 732 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 720. By way of example, and notlimitation, FIG. 11 illustrates operating system 734, applicationprograms 735, other program modules 736, and program data 737.

The computer 710 may also include other removable/non-removablevolatile/nonvolatile computer storage media. By way of example only,FIG. 11 illustrates a hard disk drive 741 that reads from or writes tonon-removable, nonvolatile magnetic media, an optical disk drive 755,and nonvolatile optical disk 756. The hard disk drive 741 is typicallyconnected to the system bus 721 through a non-removable memory interfacesuch as interface 740, and optical disk drive 755 is typically connectedto the system bus 721 by a removable memory interface, such as interface750.

Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Application-specific Integrated Circuits (e.g., ASICs),Application-specific Standard Products (e.g., ASSPs), System-on-a-chipsystems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 11, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 710. In FIG. 11, for example, hard disk drive 741 isillustrated as storing operating system 744, application programs 745,other program modules 746, and program data 747. Note that thesecomponents can either be the same as or different from operating system734, application programs 735, other program modules 736, and programdata 737.

A user may enter commands and information into the computer 710 throughinput devices such as a keyboard 762, a microphone 763, and a pointingdevice 761, such as a mouse, trackball or touch pad. Other input devices(not shown) may include a joystick, game pad, satellite dish, scanner,or the like. These and other input devices are often connected to theprocessing unit 720 through a user input interface 760 that is coupledto the system bus, but may be connected by other interface and busstructures. A visual display 791 or other type of display device is alsoconnected to the system bus 721 via an interface, such as a videointerface 790. In addition to the monitor, computers may also includeother peripheral output devices such as speakers 797 and printer 796,which may be connected through an output peripheral interface 795.

The computer 710 is operated in a networked environment using logicalconnections (such as a local area network—LAN, or wide area network—WANor a controller area network—CAN) to one or more remote computers, suchas a remote computer 780.

When used in a LAN networking environment, the computer 710 is connectedto the LAN 771 through a network interface or adapter 770. When used ina WAN networking environment, the computer 710 typically includes amodem 772 or other means for establishing communications over the WAN773, such as the Internet. In a networked environment, program modulesmay be stored in a remote memory storage device. FIG. 11 illustrates,for example, that remote application programs 785 can reside on remotecomputer 780.

It should also be noted that the different examples described herein canbe combined in different ways. That is, parts of one or more examplescan be combined with parts of one or more other examples. All of this iscontemplated herein.

Example 1 is a computing system comprising:

-   -   image receiving logic configured to receive image data        indicative of an image of a field;    -   ground identification logic configured to identify a first image        portion of the image representing ground in the field;    -   image segmentation logic configured to identify a remaining        image portion that omits the first image portion from the image;    -   crop classification logic configured to:        -   apply a crop classifier to the remaining image portion; and        -   identify a second image portion of the image that represents            locations of crop plants in the field;    -   weed identification logic configured to identify locations of        weed plants in the field based on the identification of the        first and second image portions; and    -   control signal generation logic configured to generate a machine        control signal based on the identified locations of the weed        plants.

Example 2 is the computing system of any or all previous examples,wherein the image segmentation logic is configured to obtain a thirdimage portion that omits the first and second image portions from theimage.

Example 3 is the computing system of any or all previous examples,wherein the weed identification logic is configured to identify thelocation of the weeds based on the third image portion.

Example 4 is the computing system of any or all previous examples,wherein the second image portion is identified based on spatialanalysis.

Example 5 is the computing system r of any or all previous examples,wherein the spatial analysis uses crop location data indicative of ageographic location of crops in the field.

Example 6 is the computing system of any or all previous examples,wherein the second image portion is identified based on spectralanalysis.

Example 7 is the computing system of any or all previous examples,wherein the spectral analysis uses a clustering algorithm with RGBsegmentation.

Example 8 is the computing system of any or all previous examples, andfurther comprising crop classifier training logic configured to receivetraining data and to training the crop classifier to identify the cropbased on the training data.

Example 9 is the computing system of any or all previous examples,wherein the training data comprises images labeled with crop data.

Example 10 is the computing system of any or all previous examples, andfurther comprising anomaly detection logic configured to detect ananomaly based on application of the crop classifier.

Example 11 is the computing system of any or all previous examples,wherein the crop classifier training logic is configured to train thecrop classifier based on the detected anomaly.

Example 12 is the computing system of any or all previous examples,wherein the anomaly is detected based on crop location data indicativeof a geographic location of crops in the field.

Example 13 is a method performed by a computing system, the methodcomprising:

-   -   receiving image data indicative of an image of a field;    -   identifying a first image portion of the image representing        ground in the field;    -   identifying a remaining image portion that omits the first image        portion from the image;    -   identifying, based on applying a crop classifier to the        remaining image portion, a second image portion of the image        that represents locations of crop plants in the field;    -   identifying locations of weed plants in the field based on the        identification of the first and second image portions; and    -   generating a machine control signal based on the identified        locations of the weed plants.

Example 14 is the method of any or all previous examples, and furthercomprising identifying a third image portion that omits the first andsecond image portions from the image.

Example 15 is the method of any or all previous examples, and furthercomprising:

-   -   identifying the location of the weeds based on the third image        portion.

Example 16 is the method of any or all previous examples, whereinidentifying the second image portion comprises performing spatialanalysis on the image.

Example 17 is the method of any or all previous examples, wherein thespatial analysis uses crop location data indicative of a geographiclocation of crops in the field.

Example 18 is the method of any or all previous examples, whereinidentifying the second image portion comprises performing spectralanalysis on the image.

Example 19 is a control system for an agricultural spraying machine, thecontrol system comprising:

-   -   a plant evaluation system configured to:    -   receive image data indicative of an image of a field;    -   identify a first image portion of the image representing ground        in the field;    -   define a remaining image portion that omits the first image        portion from the image;    -   identify, in the remaining image portion, a second image portion        of the image that represents locations of crop plants in the        field;    -   define a weed image that omits the first and second image        portions from the image; and    -   identify locations of weeds in the field based on the weed        image; and    -   a control signal generator configured to generate a machine        control signal based on the identified locations of the weed        plants.

Example 20 is the control system of any or all previous examples,wherein the plant evaluation system is configured to:

-   -   identify the second image portion based on at least one of        spatial analysis or spectral analysis on the image; and    -   apply a crop classifier to the second image portion.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. A computing system comprising: image receivinglogic configured to receive image data indicative of an image of afield; ground identification logic configured to identify a first imageportion of the image representing ground in the field; imagesegmentation logic configured to identify a remaining image portion thatomits the first image portion from the image; crop classification logicconfigured to: apply a crop classifier to the remaining image portion;and identify a second image portion of the image that representslocations of crop plants in the field; weed identification logicconfigured to identify locations of weed plants in the field based onthe identification of the first and second image portions; and controlsignal generation logic configured to generate a machine control signalbased on the identified locations of the weed plants.
 2. The computingsystem of claim 1, wherein the image segmentation logic is configured toobtain a third image portion that omits the first and second imageportions from the image.
 3. The computing system of claim 2, wherein theweed identification logic is configured to identify the location of theweeds based on the third image portion.
 4. The computing system of claim1, wherein the second image portion is identified based on spatialanalysis.
 5. The computing system of claim 4, wherein the spatialanalysis uses crop location data indicative of a geographic location ofcrops in the field.
 6. The computing system of claim 1, wherein thesecond image portion is identified based on spectral analysis.
 7. Thecomputing system of claim 6, wherein the spectral analysis uses aclustering algorithm with RGB segmentation.
 8. The computing system ofclaim 1, and further comprising crop classifier training logicconfigured to receive training data and to training the crop classifierto identify the crop based on the training data.
 9. The computing systemof claim 8, wherein the training data comprises images labeled with cropdata.
 10. The computing system of claim 8, and further comprisinganomaly detection logic configured to detect an anomaly based onapplication of the crop classifier.
 11. The computing system of claim10, wherein the crop classifier training logic is configured to trainthe crop classifier based on the detected anomaly.
 12. The computingsystem of claim 11, wherein the anomaly is detected based on croplocation data indicative of a geographic location of crops in the field.13. A method performed by a computing system, the method comprising:receiving image data indicative of an image of a field; identifying afirst image portion of the image representing ground in the field;identifying a remaining image portion that omits the first image portionfrom the image; identifying, based on applying a crop classifier to theremaining image portion, a second image portion of the image thatrepresents locations of crop plants in the field; identifying locationsof weed plants in the field based on the identification of the first andsecond image portions; and generating a machine control signal based onthe identified locations of the weed plants.
 14. The method of claim 13,and further comprising identifying a third image portion that omits thefirst and second image portions from the image.
 15. The method of claim14, and further comprising: identifying the location of the weeds basedon the third image portion.
 16. The method of claim 13, whereinidentifying the second image portion comprises performing spatialanalysis on the image.
 17. The method of claim 16, wherein the spatialanalysis uses crop location data indicative of a geographic location ofcrops in the field.
 18. The method of claim 13, wherein identifying thesecond image portion comprises performing spectral analysis on theimage.
 19. A control system for an agricultural machine, the controlsystem comprising: a plant evaluation system configured to: receiveimage data indicative of an image of a field; identify a first imageportion of the image representing ground in the field; define aremaining image portion that omits the first image portion from theimage; identify, in the remaining image portion, a second image portionof the image that represents locations of crop plants in the field;define a weed image that omits the first and second image portions fromthe image; and identify locations of weeds in the field based on theweed image; and a control signal generator configured to generate amachine control signal based on the identified locations of the weedplants.
 20. The control system of claim 19, wherein the plant evaluationsystem is configured to: identify the second image portion based on atleast one of spatial analysis or spectral analysis on the image; andapply a crop classifier to the second image portion.